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ABSTRACT 



A technique is provided for passively browsing the Internet 
or an intranet by displaying images from web pages on a 
user's display screen. The user can select an image by 
clicking on it using a pointer manipulated by a mouse or 
trackball. A mapping list is maintained that maps the image 
universal resource locator (URL) for each image to the URL 
of the web page containing the image. Using the mapping 
list, the user's web browser can be driven to the web page 
associated with the image selected by the user. A group of 
web pages may be defined based on the results of a search, 
by entering URLs for sites or web pages of interest, using a 
bookmarks file, based on the currently displayed web page 
in a web browser, or by determining which web pages are 
associated with the images stored in an image cache. 

20 Claims, 9 Drawing Sheets 
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METHOD FOR PASSIVELY BROWSING THE 
INTERNET USING IMAGES EXTRACTED 
FROM WEB PAGES 

This application claims the benefit of U.S. provisional 5 
application No. 60/011,435, filed Feb. 9, 1996. 

FIELD OF THE INVENTION 

This invention relates to data networks such as the Inter- 
net and intranets, and more particularly, to passively brows- 
ing for information by viewing images. 

BACKGROUND OF THE INVENTION 

A frustrating aspect of the information revolution is that is 
the ease of accessing on-line material has not kept pace with 
the growth in the amount of material that is available. For 
example, advances in hardware, compression techniques, 
editing tools, and web browsers have facilitated the creation 
of material containing inlined images (images that appear on 20 
a web page with text). Inlined images are a popular way in 
which to create eye-catching text and graphics that would be 
difficult or impossible to create using the capabilities of 
hypertext markup language (HTML). However, accessing 
web pages with interesting inlined images typically involves 25 
browsing through web pages of material by clicking on 
hypertext links. Although browsing in this way is often 
rewarding, it requires a good deal of attention on the part of 
the person who is browsing and can be burdensome. 

It is therefore an object of the present invention to provide 30 
a way in which to facilitate the access and display of Internet 
or intranet images and the material associated with the 
images. 

SUMMARY OF THE INVENTION 35 

This and other objects of the invention are accomplished 
in accordance with the principles of the present invention by 
providing a system in which a user can display web page 
images on a computer screen. When the user is interested in 
a particular image, the user can select the displayed image 
by clicking on it with a mouse or trackball. 

The system maintains a mapping list that maps the 
universal resource locator (URL) of the displayed web page 
images to the URL of the web page containing those images. 45 
When a user selects a displayed image, the user's browser is 
driven to the associated web page, so that the user can view 
the web page in its entirety. 

The group of web pages containing the images to be 
displayed can be obtained in various ways. For example, a 50 
search engine can be used to locate web pages based on 
certain search criteria. Web page or web site URLs can be 
entered manually or by using a cut and paste operation from 
an application running in an open window. A bookmarks file 
containing various URLs can also be used to define a group 55 
of web pages. Web pages can be obtained based on the URL 
for the currently displayed page in an active web browser. A 
recursive retrieval process can be used to obtain multiple 
groups of linked web pages. Regardless of how the group of 
web pages is defined, the system extracts embedded image g 0 
URLs from the web pages and forms a list mapping the 
extracted image URLs to the URLs of the web pages 
associated with those images. 

If desired, web page images can be obtained from an 
image cache. For example, in a system having a proxy server 65 
connection to the Internet, images are stored in a proxy 
server cache whenever web pages are retrieved. The proxy 
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server contains a log file that can be parsed to construct a list 
that maps images to associated web pages. 

Images may be displayed on the user's display in random 
positions, in a non-overlapping pattern, or arranged by size, 
etc. The user's client computer maintains a data list of 
coordinates so that when the user clicks on a given image, 
the client can determine which image has been selected. The 
mapping list is used to determine which web page URL 
corresponds to the selected image. The user's web browser 
is driven to the URL of that web page. 

The user can passively browse the Internet or an intranet 
by observing the displayed images. Web page images typi- 
cally contain lively text and graphics, so it is relatively easy 
for users to quickly review a large amount of material. When 
images are obtained from an image cache or from web pages 
associated with the URL for a currently displayed page of an 
active web browser, no user input is required. When the user 
submits a search or a list of web page or web site URLs, only 
minimal user input is needed. 

Further features of the invention, its nature and various 
advantages will be more apparent from the accompanying 
drawings and the following detailed description of the 
preferred embodiments. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a diagram of an illustrative display screen on 
which various web page images have been displayed in 
random positions. 

FIG. 2 is a diagram of an illustrative display screen on 
which various web page images have been displayed in a 
non- overlapping pattern. 

FIG. 3 is a diagram of a system that uses a proxy server 
in accordance with the present invention. 

FIG. 4 is a flow chart of the steps involved in using the 
system of FIG. 3 to display images and access associated 
web pages. 

FIG. 5 is a flow chart of the steps involved in saving a 
snapshot of a number of web page images. 

FIG. 6 is a diagram of a system with certain functions 
implemented using a local application in accordance with 
the present invention. 

FIG, 7 is a flow chart of the steps involved in using the 
system of FIG. 6 to display images and access associated 
web pages. 

FIG. 8 is a diagram of a system with certain functions 
implemented using processes on a server and certain func- 
tions implemented using downloaded applets in accordance 
with the present invention, 

FIG. 9 is a flow chart of the steps involved in using the 
system of FIG. 8 to display images and access associated 
web pages. 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

U.S. provisional application No. 60/011,435, filed Feb. 9, 
1996, is hereby incorporated by reference herein. 

In accordance with the present invention, a user with a 
computer connected to a data network such as the Internet or 
an intranet can have web page images 10 displayed on 
computer screen 12, as shown in FIG. 1. Typical images 
include logos, art, and pictures of products and may, in 
general, include text. If the user observes an image of 
interest, the user can select that image by clicking on the 
image by manipulating pointer 13 with a pointing device 
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such as a trackball or a mouse. When an image is selected, 
the user's web browser is automatically driven to the web 
page associated with the image. The user can browse the 
World Wide Web starting with that page. 

Various techniques can be used to define the set of images 
that are presented on screen 12. If desired, the images can be 
obtained from an image cache. The contents of the cache 
varies continually as other users browse the Web and draw 
different images into the cache. As new images are added to 
the cache, they are displayed on screen 12. Because a variety 
of images are presented to the user without any user input, 
the user may browse the Internet or intranet passively. 
Another way in which to obtain images is to perform a 
search for web pages that meet certain predefined search 
criteria. The images displayed on screen 12 can be extracted 
from the web pages that match the search criteria. Similarly, 
a user can provide a list of certain universal resource locators 
(URLs) to define a set of web pages or web sites. The URLs 
can be entered by the user manually or by cutting and 
pasting from an application or can be supplied from a web 
browser's bookmarks file. The web page images are 
obtained from the web pages associated with the list of 
URLs. Web page images can also be obtained from web 
pages associated with the currently displayed page of an 
active web browser. 

If desired, web page images 14 can be displayed on 
computer display screen 16 in a non-overlapping pattern, as 
shown in FIG. 2. The pattern of FIG. 2 uses standard 
two-dimensional bin-packing algorithms to avoid the over- 
lap associated with the random placement approach of FIG. 
1. Another suitable display technique involves displaying 
images according to size (e.g., placing the largest images in 
the center of the display). 

Images may be obtained from an image cache using 
system 18 of FIG. 3. In system 18, multiple clients are 
clustered together to form a local area network 20. Web 
servers 22 and 24, which are connected to a data network 
such as the Internet 26 or an intranet, supply web pages to 
clients 28, 30, and 32. Clients 28, 30, and 32 are connected 
to the Internet 26 through proxy server 34. Proxy servers are 
typically used in organizations that wish to provide a firewall 
machine between clients in a local area network and the 
Internet. Clients 28, 30, and 32 in local area network 20 
contain standard web browsers, such as web browser 36 on 
client 32. Each web browser is configured to use proxy 
server 34 as its proxy server. When users browse the Internet 
26, the web browsers request web pages from the proxy 
server 34. Proxy server 34 stores requested web pages and 
the images contained in or referenced from those web pages 
in cache 38. If no local copy of a requested page exists in 
cache 38 of proxy server 34, proxy server 34 retrieves the 
requested web page from the appropriate web server 22 or 
24 via the Internet 26. Proxy server 38 also maintains a 
standard log file containing the URLs of the stored web 
pages and images, 

Amontage application, such as montage application 40 on 
client 32, may run on each client in local area network 20, 
but is illustrated only as running on client 32 to avoid 
over-complicating the drawing. Montage application 40 
parses the log file maintained by proxy server 38 and creates 
a corresponding mapping list containing the URLs of the 
images in cache 38 and the URLs of the web pages asso- 
ciated with those images. Montage application 40 displays 
the images on the display of client 32 in a random pattern 
(e.g., as shown in FIG. 1), a non-overlapping pattern (e.g., 
as shown in FIG. 2), or in some other suitable arrangement. 

As montage application 40 displays the images, montage 
application 40 maintains a list of coordinates that indicate 
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the positions in which the images have been displayed. 
Montage application 40 awaits user input, so that when the 
user clicks on a given image, montage application 40 can 
determine which image has been selected and can drive web 

5 browser 36 to the web page associated with that image. 
Images displayed on client 32 are stored in display 
memory 42. If desired, a user can save a snapshot of the 
currently displayed images. Information is saved with the 
snapshot that allows the user to click on the images within 

10 the snapshot to access the web pages associated with the 
images. Snapshots are preferably stored in the form of a web 
page. If the user wishes to share the snapshot, the user can 
install the web page containing the snapshot on a publicly 
accessible server and can provide the URL for that web page 

is to interested parties. 

The steps involved in using system 18 to display web page 
images and access the web pages associated with those 
images are shown in FIG. 4. At step 44, the user configures 
web browser 36 so that proxy server 34 is identified as the 

20 proxy server for client 32. This type of configuration ensures 
that web pages accessed by the clients running the montage 
application in local area network 20 will be stored in cache 
38 and that entries will be made in a log file associated with 
proxy server 34 to indicate that the web pages have been 

25 cached. A suitable web browser 36 that allows the user to 
adjust the proxy server setup is the Navigator web browser 
of Netscape Communications Corporation. At step 46, the 
user starts the operation of montage application 40 (e.g., by 
clicking on an icon in a Microsoft Windows® environment). 

At step 48, several concurrent processes implemented by 
montage application 40 are performed on client 32. In 
process 50, montage application 40 parses the standard tog 
file maintained by proxy server 34 to construct a mapping 
list that maps the URL for each image fisted in the log file 
to the URL of the web page associated with that image. To 
construct the mapping list, the log file on proxy server 34 
must be readable by montage application 40. 

In process 52, moDtage application 40 determines whether 

4Q the images listed in the mapping file still exist in cache 38 
(i.e., the images have not been overwritten) and, if the 
images exist, places the URLs of the images on a list of 
completely retrieved images. The disk of proxy server 34 
used for cache 38 is preferably network mountable, so that 

45 montage application 40 running on client 32 can mount the 
disk (i.e., can access the image files in cache 38 as if they 
were local files on client 32). Local area network 20 pref- 
erably has standard network file system (NFS) capabilities. 
If desired, URLs that match text strings indicative of por- 

50 nographic or otherwise offensive or unwanted content can be 
filtered out during process 52. 

In process 54, montage application 40 displays the com- 
pletely retrieved images from the list on the display of client 
32 and maintains a fist of the coordinates for each displayed 

55 image. In process 56, montage application 40 waits for input 
from the user (e.g., a click on a given image). 

If a user selects an image by clicking at a certain 
coordinate, montage application 40 determines which image 
has been selected from the list of coordinates maintained 

60 during process 54 and uses the mapping list constructed 
during process 50 to drive web browser 36 to the web page 
that contains the selected image at step 58. 

If a user selects the option "create snapshot/' a snapshot 
image of the images in display memory 42 of client 32 is 

65 saved at step 60, The steps involved in creating a snapshot 
are shown in more detail in FIG. 5. The user enters the name 
of the directory where the snapshot data files are to be stored 
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at step 62. The directory can be, e.g., a local directory on a images are shown in FIG. 7. At step 90, the user selects a 

disk on client 32. At step 64, the images currently in display method for obtaining web page images. For example, a user 

memory 42 are saved as a snapshot image file in the selected at client 76 clicks on an option presented on client 76 by 

directory. The current version of the coordinate list data montage application 88. 

indicating the coordinates of the displayed images and the 5 [f the user selects the option "search," the user is provided 

URLs of the displayed images are stored in the directory as with an opportunity to enter various search parameters at 

an image map coordinate file at step 66. At step 68, a new slep 92. For example, the user can enter search terms for 

web page is created in the directory by montage application subjects of interest. Another item that the user can enter is 

40. The web page includes an image tag that specifies the t he desired depth of web page links to be searched. At step 

URL of the snapshot image file, the name of the snapshot 10 94^ a s^ch engine is run to generate one or more web pages 

image map coordinate file, and indicates that the snapshot of search results. At step 96, montage application 88 extracts 

image file is of the image map type. The web page is a hst 0 f URLs from the search results web pages generated 

installed on a suitable web server by the user or a system at step 94. If desired, a filter may be used at step 96 to ensure 

administrator at step 70 using conventional web page instal- that the extracted URLs only correspond to sites other than 

lation techniques. 15 the site of the search engine. This type of filter excludes links 

If desired, the user can "send" the snapshot to a friend, by to commercial entities associated with the search engine 

communicating the URL of the snapshot web page to the service (e.g., links to Alta Vista services or Digital Equip - 

friend (e.g., by e-mail, etc.). Because the snapshot has been ment Corporation when the Alta Vista search engine of 

installed on a web server, it may be accessed by anyone with Digital Equipment Corporation is being used), 

the appropriate URL. When the accessing user clicks on a 20 [f the user se]ects the option « web ^ „ montage appli . 

region of the snapshot image map, the user's web browser ca ti on 88 provides the user the opportunity to enter URLs for 

sends the coordinates of the user's selection to the web web pages or web sites and a desired depth at step 98. URLs 

server along with the name of the image map coordinate file. can be entered manually or may be entered with a cut and 

The web server determines the URL of the web page paste operation from an active application. If desired, step 

associated with the selected region by searching through the 25 98 can involve automatically deriving the URL for the 

image map coordinate file for the first region that contains currently displayed page of an active web browser. The URL 

the selected coordinates. The web server then sends back a can be obtained using a web browser with an Application 

command that drives the user's web browser to the web page Programmer Interface (API) that allows montage applica- 

associated with the image in the selected region. tion 8g to query the web browser to determine the currently 

In system 18 of FIG. 3, the various clients connected to 30 displayed page. Alternatively, the code for a web browser 

proxy server 34 define a community of users. The web pages -;■ can be altered, so that the web browser provides montage 

and images retrieved by the community of users are reflected application 88 with the URL of its currently displayed page, 

in the contents of cache 38. Because montage application 40 If desired, web browser 84 can be configured to use a 

displays the images stored in cache 38, the displayed images dummy proxy server process (e.g., on client 76) to allow the 

mirror the interests of the community of users. 35 URL of the current page to be ascertained by montage 

Another way in which images can be displayed involves application 88. 

performing a search or entering web site information. In If the user selects the option "bookmarks," montage 

system 72, clients 74 and 76 are interconnected with web application 88 presents the user with the opportunity to enter 

servers 78 and 80 via the Internet 82, as shown in FIG. 6. 4Q a desired depth and the name of various bookmarks files at 

The group of web images that are displayed on clients 74 step 100. Step 100 may also involve the conversion of the 

and 76 may be based on a search, a list of URLs, a bookmarks files into a list of URLs, if necessary, 

bookmarks file specified by the user, or the currently dis- Several concurreil t processes are performed at step 102. In 

played page of an active web browser. process 104, montage application 88 uses the list of URLs 

Client 76 has standard web browser 84, standard display 45 and the desired depth to retrieve the group of web pages 

memory 86, and montage application 88. Additional clients, from which images are extracted. If the desired depth is 0, 

such as client 74, may have the same configuration as client only the web pages corresponding to the URLs on the list are 

76, but this detail is not included in FIG. 6 to avoid retrieved. If the desired depth is greater than 0, the web page 

over-complicating the drawing. Montage application 88, retrieval process is iterated as needed. For example, if the 

which controls the display of web page images and the 50 depth is 1, web pages corresponding to links appearing on 

access of corresponding web pages, may be implemented in the web pages for the depth of 0 case are retrieved in 

the Java programming language of Sun Microsystems. If addition to the web pages for the depth ofO case. If the depth 

montage application 88 is implemented in Java, it is run with is 2, an additional layer of web pages is retrieved. As the web 

a Java interpreter installed on client 76. With the arrange- pages are retrieved, montage application 88 identifies the 

ment of FIG. 6 and standard Windows® software, the user 55 image URLs that are embedded in the web pages and 

at client 76 can run Montage application 88 in one window constructs a mapping that associates each image URL to the 

and web browser 84 in another window. Because system 72 URL of the web page that contains that image, 

does not have a proxy server for client 76, system 72 may Id addition, the web pages that are retrieved can be filtered 

be somewhat slower in performing certain web page dm ' mg process 104. For example, montage application 88 

retrieval operations than system 18 (FIG. 3). However, 60 might on i y retairj web pages from the site as 

system 72 does not require clients to be configured in a entered by the user to ensure that the retrieved web pages are 

communal arrangement with a single proxy server. In simila r in content to the initially entered search. This filter 

addition, system 72 does not require client processes to be can be used in conjunction with the type of filter used at step 

able to access log files and data files associated with a proxy 96 (which excludes "same site" links). For example, during 

server cache. 65 me f ew i ayers 0 f web page retrieval the filter can exclude 

The steps involved in using system 72 to display web page links to sites other than for the listed URLs, thereby ensuring 

images and access the web pages associated with those that the retrieval process stays focused and is able to proceed 
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past the initial menu pages common with many sites. After 
these first few layers, the filter can exclude links to the same 
site, thereby maximizing the breadth of the search at this 
stage. The user can enter the depth for each part of the web 
page retrieval filtering process. 

In process 106, the image for each image URL in the 
mapping list is retrieved. If desired, images can be excluded 
during image retrieval process 106 if they correspond to web 
page URLs that match text strings indicative of porno- 
graphic or otherwise offensive or unwanted content. When- 
ever an image has been completely retrieved, that image is 
placed on a list of completely retrieved images. 

In process 108, montage application 88 displays the 
completely retrieved images from the list on the display of 
client 76 and maintains a list of the coordinates for each 
displayed image. To display images on the display, image 
data is placed in display memory 86. In process 110, 
montage application 88 monitors user input. 

If a user selects an image by clicking at a coordinate 
within the image, montage application 88 determines the 
image URL for the selected image from the list of image 
coordinates maintained in process 108 and uses the mapping 
list constructed in process 104 to drive web browser 84 to 
the web page that contains the selected image at step 114. 

If a user selects the option "create snapshot," a snapshot 
image of the images in display memory 86 of client 76 are 
saved at step 112. The step of creating the snapshot involves 
the steps described in connection with FIG. 5. 

Another arrangement that can be used is shown in FIG. 8. 
In system 116, clients 118, 120, and 122 are interconnected 
with web servers 124, 126, and 128 via the Internet 130 or 
an intranet. Web servers 126 and 128 provide web pages of 
Internet material. Web server 124 contains applets that may 
be downloaded by clients 118, 120, and 122 when it is 
desired to use the montage process. The downloaded applets 
can be combined with standard web browsers to form web 
browser with montage applet 132 and web browser with 
montage applet 134. The operation of web server 124 is 
controlled by montage server process 136 and image server 
process 138. The group of web images that are displayed on 
the clients by montage applets 132 and 134 may be obtained 
from image cache 140 or may be based on a search, a list of 
URLs, or a bookmarks file specified by the user. 

Clients 118, 120, and 122 have standard display 
memories, such as display memories 142 and 144. Web 
browser with montage applet 132 and web browser with 
montage applet 134 (hereinafter montage applets) control 
the display of web page images and the accessing of 
corresponding web pages and are preferably implemented in 
the Java programming language of Sun Microsystems. The 
arrangement of FIG. 8 allows an interested user to download 
and run the code for the montage process without having a 
Java interpreter. In addition, the arrangement of FIG. 8 
allows clients 118, 120, and 122 to be geographically remote 
from each other and from web server 124. 

The steps involved in using system 116 to display web 
page images and access the web pages associated with those 
images are shown in FIG. 9. At step 150, web server 124 
provides an initialization web page, which may be retrieved 
by the user at, e.g., client 122 using a standard web browser. 

At step 152, the user fills in blank fields in the initializa- 
tion web page. For example, the user may be asked to supply 
credit card and billing information before being allowed to 
proceed. The user also selects the desired method for obtain- 
ing images. If the user selects the option "cache/' the images 
are obtained from image cache memory 140. Images are 
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drawn into cache 140 when users at clients 118, 120, and 122 
use montage server process 136 on web server 124 to 
retrieve web pages with images (e.g., from web servers 126 
and 128). 

5 Montage applet 134 is downloaded to the client 122 at 
step 154 where it is automatically run by the web browser 
portion of web browser with montage applet 134. Several 
concurrent processes are performed at step 156. In process 
158, montage server process 136 maintains a mapping list 

10 that associates image URLs with the URLs of the web pages 
that contain the images. In process 160, montage server 
process 136 and image server process 138 determine 
whether the images listed in the mapping file still exist in 
cache 140 (i.e., the images have not been overwritten) and, 

15 if the images exist, places the URLs of the images on a list 
of completely retrieved images. The completely retrieved 
images are transmitted to montage applet 134 on client 122 
by montage server process 136 during process 162. Undes- 
ired URLs are filtered out during process 162 by comparing 

20 the text of the URLs to text strings provided by the user at 
client 122 that are indicative of pornographic or otherwise 
offensive or unwanted content. Images that correspond to 
. undesired URLs are not transmitted to client 122 during 
process 162. 

25 Montage applet 134 displays the transmitted images on 
the display of client 122 and maintains a list of the coordi- 
nates for each displayed image during process 164. In 
process 166, montage applet 134 waits for user input (e.g., 
in the form of a click on a given image). 

30 if a user selects an image by clicking at a certain 
coordinate, montage applet 134 determines which image has 
been selected using the list of image coordinates maintained 
during process 164 and uses the mapping list maintained 
during process 158 to drive web browser with montage 

35 applet 134 to the web page containing the selected image at 
step 168. 

If a user selects the option "create snapshot" during step 
156, a snapshot image of the images in display memory 144 
of client 122 are saved at step 170. The steps involved in 

40 creating a snapshot are described in connection with FIG. 5. 
If the user selects the option "search" at step 152, the user 
is given an opportunity to enter search terms and a desired 
depth value at step 172. The search terms and depth are 
preferably entered as part of the initialization information 

45 provided by the user. Step 174 involves downloading mon- 
tage applet 134 from web server 124 to client 122. At step 
176, a search engine is run by montage server process 136 
based on the search terms supplied by the user. The search 
generates one or more web pages of search results. At step 

50 178, montage server process 136 extracts a list of URLs 
from the search results web pages generated at step 176. If 
desired, a filter may be used at step 178 to ensure that the 
extracted URLs only correspond to sites other than the site 
of the search engine. Such a filter excludes links to com- 

55 mercial entities associated with the search engine service. 
If the user selects the option "web site" at step 152, the 
user is given an opportunity to enter information concerning 
web sites and web pages and a desired depth value as part 
of the initialization information, as shown by step 180. If 

60 desired, the user can enter a URL at step 180 from the 
currently displayed web page of a web browser. The web site 
and web page URLs entered by the user form a list of URLs. 
Montage applet 134 for client 122 is downloaded at step 
182. 

65 Several concurrent processes are performed at step 184. 
During process 186, server process 136 retrieves web pages 
based on the list of URLs obtained from step 178 or 180 and 
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the desired depth. Process 186 is an iterative process. If the 
desired depth is 0, only the web pages corresponding to the 
URL(s) on the list are retrieved. If the desired depth is 
greater than 0, additional layers of web pages are retrieved. 
For example, if the depth is 1, web pages corresponding to 5 
links appearing on the web pages for the depth of 0 case are 
retrieved in addition to the web pages for the depth of 0 case. 
If the depth is 2, a further layer of web pages is retrieved. As 
the web pages are retrieved, montage server process 136 
identifies the image URLs that are embedded in the web 
pages and constructs a mapping list that maps each image 10 
URL to the URL of the web page associated with that image. 

In process 188, montage server process 136 passes the 
image URL for each image to image server process 138, 
which retrieves the image. When the image has been fully 
retrieved, image server process 138 notifies montage server 15 
process 136 and stores the image in image cache 140. In 
process 190, montage server process 136 transmits the 
completely retrieved images to montage applet 134 on client 
122. If desired, images corresponding to web page URLs 
that match text strings indicative of pornographic or other- 2Q 
wise offensive or unwanted content can be excluded during 
process 190. Before transmitting the images, montage server 
process 136 obtains the user's criteria for undesired URLs 
(e.g., as part of step 152). Images corresponding to undes- 
ired URLs are then filtered out in process 190. 

In process 192, montage applet 134 displays the trans- 25 
mitted images on the display of client 122 and maintains a 
list of the coordinates for each displayed image. To display 
images on the display, image data is placed in display 
memory 144. During process 194, montage applet 134 waits 
for input from the user. 30 

If a user selects an image by clicking on a coordinate 
within the image, montage applet 134 determines the image 
URL for the selected image from its list of image coordi- 
nates maintained during process 192 and uses the mapping 
list constructed during process 186 to drive the web browser 35 
on client 122 to the web page that contains the selected 
image at step 168. 

If a user selects the option "create snapshot" during step 
184, a snapshot image of the images in display memory 144 
of client 122 are saved at step 170. Creating the snapshot 40 
involves steps described in more detail in connection with 
FIG. 5. 

With the system of FIGS. 8 and 9, images are drawn into 
image cache 140 whenever a user at a client uses montage 
server process 136 to initiate a search or enter URLs to 45 
define a group of web pages. To make this type of arrange- 
ment captivating for users, web servers such as web server 
124 may be dedicated to particular subjects. With such an 
arrangement, the users who view images from the image 
cache of a given server will be able to quickly view images 
that are primarily associated with the subject matter of that 50 
server. 

With the proxy server cache system of FIGS. 3 and 4, the 
community of users connected to proxy server 34 defines the 
type of images that are contained in cache 38. 

If desired, the images supported by the montage system 55 
can be moving images, such as digital video files or digital 
animation files. The display of the images can be initiated by 
the user or may be invoked automatically. For example, 
images from an image cache or based on a default search or 
URL list could be displayed in place of a screen saver. When 60 
a certain amount of time (e.g., 5 minutes) has elapsed 
without any input from the user, the images are placed on the 
display. 

The foregoing is merely illustrative of the principles of 
this invention and various modifications can be made by 65 
those skilled in the art without departing from the scope and 
spirit of the invention. 



What is claimed is: 

1. A method for use in a communications network in 
which a plurality of users at clients are connected to a data 
network, wherein the users access web pages that have 
associated web page images and are stored on servers 
connected to the data network, the method comprising the 
steps of: 

obtaining a plurality of the images, each image being 
associated with one of the web pages, wherein the step 
of obtaining the images further comprises the substeps 
of, 

retrieving a group of web pages based on a list of 
universal resource locators (URLs), and 

retrieving the images associated with the group of web 
pages; 

displaying the images on one of the clients without 

displaying the associated web pages; 
allowing the user to select an image from among the 

displayed images; and 
accessing the web page associated with the selected 

image; 

wherein each web page of said group of web pages is 
included in said list of universal resource locators or 
can be accessed by following links from one of the 
universal resource locators in said list of universal 
resource locators. 

2. A method for use in a communications network in 
which a plurality of users at clients are connected to a data 
network, wherein the users access web pages that have 
associated web page images and are stored on servers 
connected to the data network, the method comprising the 
steps of: 

obtaining a plurality of the images, each image being 
associated with one of the web pages, wherein the step 
of obtaining the images further comprises the substeps 
of, 

retrieving a group of web pages based on the universal 
resource locator (URL) of a web page currently 
displayed in a web browser, and 

retrieving the images associated with the group of web 
pages; 

displaying the images on one of the clients without 

displaying the associated web pages; 
allowing the user to select an image from among the 

displayed images; and 
accessing the web page associated with the selected 

image; 

wherein each web page of said group of web pages can be 
accessed by following links from said web page cur- 
rently displayed in the web browser. 

3. The method defined in claim 2 wherein the step of 
displaying the images comprises the step of displaying 
images in random positions. 

4. The method defined in claim 2 wherein the step of 
displaying the images comprises the step of displaying 
images in a non-overlapping pattern. 

5. The method defined in claim 2 wherein the step of 
allowing the user to select an image from among the 
displayed images comprises the step of allowing a user to 
click on a displayed image. 

6. The method defined in claim 2 wherein the step of 
accessing the web page associated with the selected image 
comprises the steps of: 

determining the URL of the web page associated with the 

selected image; and 
driving a web browser to that URL. 
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7. The method defined in claim 2 further comprising the 
step of creating a snapshot of the displayed images. 

8. The method defined in claim 2 wherein the data 
network is the Internet, the servers are web servers, and the 
images are inlined images. 5 

9. A system for use in a communications network in which 
a plurality of users at clients are connected to a data network, 
wherein the users access web pages that have associated web 
page images and are stored on servers connected to the data 
network, the system comprising: 10 

means for obtaining a plurality of the images, each image 
being associated with one of the web pages, wherein 
the means for obtaining the images includes, 
means for retrieving a group of web pages based on a 

list of universal resource locators (URLs), and 15 
means for retrieving the images associated with the 
group of web pages; 

means for displaying the images on one of the clients 
without displaying the associated web pages; 

means for allowing a user to select an image from among 20 
the displayed images; and 

means for accessing the web page associated with the 
selected image; 

wherein each web page of said group of web pages is 
included in said list of universal resource locators or 25 
can be accessed by following links from one of the 
universal resource locators in said list of universal 
resource locators. 

10. A system for use in a communications network in 
which a plurality of users at clients are connected to a data 30 
network, wherein the users access web pages that have 
associated web page images and are stored on servers 
connected to the data network, the system comprising: 

means for obtaining a plurality of the images, each image 
being associated with one of the web pages, wherein 35 
the means for obtaining the images includes, 
means for retrieving a group of web pages based on the 
universal resource locator (URL) of a webpage cur- 
rently displayed in a web browser, and 
means for retrieving the images associated with the 40 
group of web pages; 

means for displaying the images on one of the clients 
without displaying the associated web pages; 

means for allowing a user to select an image from among 
the displayed images; and 45 

means for accessing the web page associated with the 
selected image; 

wherein each web page of said group of web pages can be 
accessed by following links from said web page cur- 
rently displayed in the web browser, 

11. The system defined in claim 9 wherein the means for 
displaying the images comprises means for displaying 
images in random positions. 

12. The system defined in claim 9 wherein the means for ^ 
displaying the images comprises means for displaying 
images in a non-overlapping pattern. 

13. The system defined in claim 9 wherein the means for 
allowing the user to select an image from among the 
displayed images comprises means for allowing a user to 
click on a displayed image. 

14. The system defined in claim 9 wherein the means for 
accessing the web page associated with the selected image 
comprises: 

means for determining the URL of the web page associ- fi5 

ated with the selected image; and 
means for driving a web browser to that URL. 
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15. The system defined in claim 9 further comprising 
means for creating a snapshot of the displayed images. 

16. The system defined in claim 9 wherein the data 
network is the Internet, the servers are web servers, and the 
images are inlined images. 

17. A method for use in a communications network in 
which a plurality of users at clients are connected to a data 
network, wherein the users access web pages that have 
associated web page images and are stored on servers 
connected to the data network, the method comprising the 
steps of: 

obtaining a plurality of the images, each image being 
associated with one of the web pages, wherein the step 
of obtaining the images further comprises the substeps 
of, 

retrieving a group of web pages based on a list of 
universal resource locators extracted from the cache 
of web pages on a proxy server, and 

retrieving the images associated with the group of web 
pages; 

displaying the images on one of the clients without 

displaying the associated web pages; 
allowing the user to select an image from among the 

displayed images; and 
accessing the web page associated with the selected 

image; 

wherein each web page of said group of web pages is 
included in said list of universal resource locators or 
can be accessed by following links from one of the 
universal resource locators in said list of universal 
resource locators. 

18. A system for use in a communications network in 
which a plurality of users at clients are connected to a data 
network, wherein the users access web pages that have 
associated web page images and are stored on servers 
connected to the data network, the system comprising: 

means for obtaining a plurality of the images, each image 
being associated with one of the web pages, wherein 
the means for obtaining the images includes, 
means for retrieving a group of web pages based on a 
list of universal resource locators extracted from the 
cache of web pages on a proxy server, and 
means for retrieving the images associated with the 
group of web pages; 
means for displaying the images on one of the clients 

without displaying the associated web pages; 
means for allowing a user to select an image from among 
the displayed images; and 

means for accessing the web page associated with the 
selected image; 

wherein each web page of said group of web pages is 
included in said list of universal resource locators or 
can be accessed by following links from one of the 
universal resource locators in said list of universal 
resource locators. 

19. The method of claim 1 wherein the step of retrieving 
a group of web pages employs a depth characteristic to 
control the retrieval process. 

20. The method of claim 2 wherein the step of retrieving 
a group of web pages employs a depth characteristic to 
control the retrieval process. 
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METHOD AND APPARATUS FOR QUERY 
REFINEMENT 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates generally to the field of 
querying a database of images and particularly to providing 
an interface for organizing the results of queries applied to 
a database of images. 

2. Description of the Related Art 

Search engines are some of the most popular and widely 
accessed sites on the World Wide Web (WWW) because 
they allow a user to find information of interest. These 
engines gather textual information about resources on the 
Web and build up indices. The indices allow for the retrieval 
of documents containing user specified keywords. Another 
method of searching for information on the Web utilizes 
subject-based directories, which provide a useful browsable 
organization of information. 

However, with the explosive growth of information that is 
available on the WWW, most queries applied to the search 
engines result in a large number of retrieved documents. 
This leads to a key limitation suffered by most of the popular 
search engines available today. More specifically, most 
search engines show the results of a query as pages of 
scrolled lists. If a large number of documents are retrieved, 
scrolling through such lists to find documents of interest to 
the user is extremely tedious and not at all user friendly. 

An image search engine, called Advanced Multimedia 
Oriented Retrieval Engine (AMORE), developed by the 
present inventors at NEC, allows the user to specify an 
image and find images that are visually similar to it. See 
Mukhetjea et al., "Towards a Multimedia World-Wide Web 
Information Retrieval Engine," Proceedings of the Sixth 
International World-Wide Web Conference, pages 177-188, 
April, 1997. In AMORE the user can specify keywords, 
where all the images relevant to the keywords are retrieved. 
However, as more and more images are indexed, AMORE is 



25 



30 



pages 318-329, June, 1992. Once again, this effort focuses 
on browsing a text collection, rather than an image collec- 
tion. 

In some instances, clustering has been used for searching 
large image databases as a pre-processing step to filter out 
non-interesting images. This reduces the number of images 
to be processed by more computer-intensive fine matching 
procedures. In addition, using the classification results pro- 
duced by clustering, users can roughly grasp the contents of 
the entire image database and use this information for their 
interaction with the image retrieval system. Image clustering 
can be based on various image features such as colors, 
textures, shapes, etc. For example, Hirata et al., "The 
Concept of Media-based Navigation and Its Implementation 
on Hypermedia System "Miyabi,"" NEC Research and 
Development, 35(4) :4 10-420, 1994, discusses a technique 
that focuses on color information in an image. Color values 
firom the image are extracted and mapped on the HLS color 
spaces. Based on the results, users can filter out the images 
before searching. On the other hand, the work described in 
Del Bimbo et al., "Shape Indexing by Structural Properties/' 
International Conference on Human Factors in Computing 
Systems, page 370-377, June, 1997, clusters by shape 
similarity. Using multi -scale analysis, a hierarchical struc- 
ture of shape is extracted. Based on this structure, searching 
capabilities are provided. However, this method assumes 
that boundary analysis correctly extracts the boundaries of 
objects in the image. However, this is very difficult to do for 
WWW images containing many objects. Similarly, image 
clustering using feature vectors based on moment invariant 
(See Flickner et al., "Query by Image and Video Content: 
The QBIC System," IEEE Computer, 28(9):23-31, 
September, 1995) or boundary features (See Mehrotra et al., 
"Similar-Shape Retrieval in Shape Data Management," 
IEEE Computer, pages 57-62, September, 1995) also 
assumes the correct extraction of objects (or input by a user). 
Therefore, it is very hard to apply these techniques to the 
WWW. Additionally, each of these techniques focuses on 
providing only a single cluster mechanism. The techniques 



suffering from the same problems affecting traditional text 4 o do n Qt address providing a user-friendly interface. 



search engines. More specifically, a query presented by the 
user of the search engine may result in a large number of 
retrieved images. Many user queries result in more than 100 
retrieved images. As with other search engines, AMORE 
shows a few of the retrieved images per page and allows the 45 
user to go to other pages of retrieved images. This is shown 
in FIG. 1. Obviously, this is not a very user-friendly way of 
displaying search results to the user of the search engine. 

Various searching and clustering techniques have been 
proposed to attempt to make systems more user-friendly. 
However, most have focused on text-based mechanisms. For 
example, clustering has been an emerging information 
retrieval tool. See Sal ton et al., "Introduction to Modern 
Information Retrieval," McGraw-Hill, 1983; Van- 
Rijsbergen, "Information Retrieval," Butterworths, 1979. 
The focus has been on the use of text clustering to speed up 
the searching process. Instead of comparing all documents 
in a large collection to find the relevant documents, such 
efforts have focused on initially comparing only a represen- 
tative cluster vector for each cluster with the search request. 
Thereafter, only documents from clusters which matched 
favorably are compared. Clustering is also useful for brows- 
ing large online collections of text as shown by a scatter/ 
gather technique. See Cutting et al., "Scatter/Gather: A 
Cluster-based Approach to Browsing Large Document 
Clusters," Proceedings of the ACM SIGIR '92 Conference 
on Research and Development in Information Retrieval, 
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The search techniques are also being applied to the 
WWW. Research at Xerox Palo Alto Research Center 
(PARC) uses clustering to extract useful structures from the 
WWW. See Pirolli et al., "Silk from a Sow's Ear: Extracting 
Usable Structures from the Web," Proceedings of the ACM 
SIGCHI '96 Conference on Human Factors in Computing 
Systems, pages 118-125, April, 1996; Pitkow et al, "Life, 
Death and Lawfulness on the Electronic Frontier," Proceed- 
ings of the ACM SIGCHI '97 Conference on Human Factors 
in Computing Systems, pages 383-390, March, 1997. The 
clusters are determined by various criteria like co-citation 
analysis. Clustering by textual similarity to organize the 
results of Web text search engines is described in Chang et 
al., "Customizable Multi-Engine Search Tool with 
Clustering," Proceedings of the Sixth International World- 
Wide Web Conference, pages 257-264, April, 1997. 
However, such implementations also fail to provide a user 
friendly user interface. 

Examples of interfaces which have been developed for 
viewing information retrieval results include SenseMaker 
(See Baldonado et al., "SenseMaker: An Information- 
Exploration Interface Supporting the Contextual Evolution 
of a User's Interest," Proceedings of the ACM SIGCHI '97 
Conference on Human Factors in Computing Systems, 
pages 11-18, March, 1997) and Cat-a-Cone (See Hearst et 
al., "Cat-a-Cone: An Interactive Interface for Specifying 
Searches and Viewing Retrieval Results using a Large 
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Category Hierarchy/' Proceedings of the ACM SIGIR '97 FIG. 8 shows the results of an object extraction procedure 

Conference on Research and Development in Information used to categorize images by objects. 

Retrieval, July, 1997). These articles, as well as those pjcs. 9(a) and 9(b) shows templates used to cluster 

discussed above are hereby incorporated herein by images by primary object and also shows the clustering 

reference.). 5 procedure. 

SenseMaker is an interface for ^formation exploration piGS ^ ^ 1Q(/)) shown search 

across heterogeneous sources in a digital library. Sense- . w . « • . « ... . r 

Maker allows the user to interactively change the level of according to primary object, along with the images for one 

granularityandtheorganizadonaldimensionsofacoUection of thc S rou P s >. lD Wlth an embodiment of the 

of documents that match the user's interest. Cat-a-Cone is a 30 P resent inve ntion. 

3D interface that allows a user to search and browse a large DETAILED DESCRIPTION OF THE 

category hierarchy and the associated documents. The docu- INVENTION 
ments and the categories can be viewed concurrently. 

However, as with other systems, SenseMaker and Cat-a- A preferred embodiment of a method and apparatus for 

Cone both focus on the text properties of documents and also 5 providing an interface for refining a query, according to the 

present differing text organizational structures. present invention, is described below in detail with reference 

Other Web Image Search engines, such as Excalibur's <° the accompanying drawings. It is to be noted that while 

Image Surfer (http://isurf.yahoo.com) and Webseer (http:// following discussion is presented m the context of the 

webseer.cs.uschicago.edu) show retrieved images in succes- AMORE search engine, the present invention is not so 

sive HTML pages. However, no technique for organizing the 20 hmited * ^ P re f nt invention may be applied to a variety of 

search results or query refinement is possible. Other popular search en g mes for me rmildmedia databases, etc. 

text search engines, such as Alta Vista (http:// A brief overview of how a search engine, such as 

www.altavista.digital.com) and Excite (http:// AMORE, operates is first presented. Implementing a search 

www.excite.com) allow query refinement. Excite can group engine involves two phases, indexing and querying. Several 

search results by Web sites. Moreover, both Excite and Alta 25 systems exist that allow full-text indexing of Web sites. The 

Vista use co-occurrence analysis to show other relevant indices allow the user to search the indexed sites using any 

keywords for the query terms. While the technique is useful, combination of keywords. Systems that can index images in 

there is no mechanism to find out the actual documents that a database based on various image characteristics, such as 

correspond to the different keywords. The keywords are sha P e and color have also been developed. Image indexing 

merely used to refine a search by including or excluding 30 allows for toe retrieval of images in the database that are 

them. Allowing the user to see the actual documents/images similar to a user specified image. AMORE integrates the two 

that belong to a cluster would be more useful. types of systems by allowing for the retrieval of Web 

documents containing one or more keywords and/or images 

SUMMARY OF THE INVENTION similar to a ^ spccificd image . FIG. 2 shows the results of 

To overcome these and other difficulties, the present 35 a search in which an integrated keyword and image query 

invention is directed to a method and apparatus for provid- was issued. 

ing an interface for query refinement. The present invention The ^ step to implementing a search engine involves 
provides to a user of a search engine a Query Result indexing the database of Web sites, documents or images. 
Visualization Environment (QRVE), a user interface which After text and/or image indices have been built, users can 
allows the user to refine the results of an original query by 4Q issue queries. The queries can contain one or more keywords 
using variously clustered results of the original query. The or an image. The text and image search can also be corn- 
results may be clustered using text, a primary object (or bined by using Boolean logic (e.g. "and" or "or"). However, 
shape) or universal resource locator (URL). The members of a problem arises when the database to be searched becomes 
the clusters may then be used to formulate the refined very large. Queries may result in a large number of retrieved 
queries. 45 images. 

BRIEF DESCRIPTION OF THE DRAWINGS The present invention provides an interface to the user of 

Hie above objects and advantages of the present invention a search engine for refining queries. The interface^ called a 

will become more apparent by describing in detail preferred ^ v ^zation Environment (QRVE), allows 

embodiments thereof with reference to the attached draw- ^ iwulis of a search to be organized m various ways. For 

ines in which* 50 exam Pte» tne images retneved as a result of a search query 

^ , ' * u f «_ j-.- i u • *u can be grouped or clustered by the Web sites (URL) from 

FIG. 1 shows the results of a traditional search using the , . , f j u u i j <i_ 

A . mnr , ° which they were retneved, by how many keywords they 

AMORE search engine. « 7 u *u • -s i.- 

— _ „ , * , , . , . , have m common, or by then- composition, where images 

ke ord ue * 11MgC wit ° similar composition form a single cluster. 

C ^™°\ query. ... 55 In accordance with the present invention, when a user 

RG. 3 shows the architecture of the system implementing ^ a quc ^ ^ user has mc opt{on of grQuping ^ fesuUs 

the present invention. in Qne of scveraJ wayS) for cxampl ^ by URLj text (or 

FIGS. 4(a) and 4(b) show search results grouped accord- key words) or composition. The QRVE is a graphical user 

ing to URL and an image from a selected URL in accordance interface (GUI) presented to the user of a system upon the 

with an embodiment of the present invention. 60 execut ion of a query. The QRVE presents to the user the 

FIGS. 5(a) and 5(6) show the hierarchical structure of a cluster information for the retrieved images, where the 

web site and images from a selected section of the web site. are clustered or grouped in accordance with the 

FIG. 6 shows the text cluster formation technique in grouping method selected by the user. Each cluster is 

accordance with an embodiment of the present invention. represented in the QRVE by a glyph (graphical element). 

FIGS. 1(a) and 1(b) show search results grouped accord- 65 When the user requests the details of a cluster, by clicking 

ing to keyword, along with the images for one of the groups, on the glyph representing the cluster, the images of the 

in accordance with an embodiment of the present invention. cluster are shown to the user. The clustered search results of 
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the initial query issued by the user may then be used to refine 
the search. The details of refinement will be discussed 
below. 

The architecture of a system upon which the present 
invention may be implemented is shown in FIG. 3. After the 5 
AMORE search engine completes indexing the images in 
the keyword and image indices, the cluster generator uses 
the keyword and image clustering algorithm to generate the 
text and image clusters, respectively. The QRVE Facilitator 
retrieves the cluster information for images retrieved in 1Q 
response to a query. This information is used to create the 
visualization, where each cluster is represented by a glyph. 
When the user clicks on a glyph representing a particular 
cluster, a request is sent by the QRVE applet to the facili- 
tator. The facilitator interacts with the text and image 
clusters to determine the images for the cluster clicked by 15 
the user. Those images are shown in the results frame of the 
client browser. 

The details of each of the methods of clustering will now 
be discussed. First, grouping by URL will be discussed. 
Suppose a user is interested in a picture of a baseball player 20 
whose last name is "Jordan." Unfortunately, she does not 
remember his first name. If a keyword search is initiated 
with the last name "Jordan" in the sports category of a search 
engine, a large number of images is retrieved. Not 
surprisingly, most of the retrieved images are of basketball 25 
star Michael Jordan. Browsing through all of the retrieved 
images to find the one desired by the user can be painful. The 
QRVE of the present invention provides a solution, as shown 
in FIGS. 4(a) and 4(fc), which show two parts of a window. 
Initially, it is assumed that the user chose the method of 30 
clustering the query search results by URL. The window in 
FIG, 4(a) shows the search results categorized by URLs 
clusters for the images retrieved. Each URL cluster is 
represented by a glyph. The QRVE for the keyword only 
search using the term "Jordan" shows that images from 35 
seven Web sites were retrieved. The desired image of the 
baseball player will most probably be in the web site 
www.majorleaguebaseball.com. When the user clicks on the 
glyph associated with this web site, the retrieved picture of 
baseball player Brian Jordan is shown in FIG. 4(6). The user 40 
can then initiate a refined search using this image of Brian 
Jordan as the query. 

The hierarchical structure of web sites from which images 
are retrieved may also be shown to the user. This is espe- 
cially useful if a large number of images from one particular 45 
web site arc retrieved. For example, for the keyword only 
search using the term "Jordan", many images of Michael 
Jordan are retrieved from the web site www.nba.com. When 
the user clicks on the glyph associated with this web site, the 
images and the directory structure of the web site are shown 50 
in a results frame. If the user is only interested in images 
from the 1997 NBA finals, she can click on the glyph for the 
subdirectory finals 97, The resultant scenario is shown in 
FIGS. 5(c) and 5(b), which together comprise a window. 
Note that in FIG. 5(a) the glyph is a circle on the path chosen 55 
by the user and is a square for other paths. The size of the 
glyph is proportional to the number of images retrieved in 
the cluster. The cluster of images is shown in FIG. 5(b). 
Additionally, the color of the glyph may be comprised of 
two basic color components, where the magnitude of the first 60 
color component is proportional to the maximum number of 
keywords in the documents containing the retrieved images 
of the group which is represented by the glyph. The mag- 
nitude of the second color component is proportional to the 
maximum image similarity. 65 

If images from many unknown web sites are retrieved, or 
if a search is simply being performed on a database of 
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images, clustering by URL is not very useful. Instead, 
clustering by keywords may be used. The text clustering 
procedure is explained as follows. 

Several preprocessing steps are carried out. First, during 
an index phase, various heuristics are used to assign key- 
words to images. The image name, the title of the page from 
which the image came, and a section of the text before and 
after the image are assigned to the image. 

Second, the documents are represented in a vector space 
model. For example, let V be the set of unique words 
assigned to the images of a category. Then each image can 
be represented by a vector of length | V| in which the absence 
of a word is indicated by 0 and the presence of a word is 
indicated by a positive number, known as term weight. The 
term weight reflects the importance of the word for the 
image. Traditional text retrieval using Inverse Document 
Frequency as the term weight has been found to be quire 
effective. See Salton et al., "Introduction to Modem Infor- 
mation Retrieval," McGraw-Hill, 1983. In the present 
invention, the term weight of a word is determined not only 
by its frequency but also by where it occurs. Thus, if a word 
occurs in the name of the image itself, its term weight will 
be greater than for a word occurring in the text surrounding 
the image. Note that before forming the vectors, universal 
common words, such as "and" and "the" are removed, as 
well as words that are very common (occurring in more than 
half the documents) for the category (like the terms "game" 
and "team" in the "sports" category). 

Third, a similarity matrix is formed by calculating the 
similarity between each pair of images. The similarity 
between two images is measured by the cosine of the angle 
between the vectors representing them. This is normalized 
so that, for each pair of images, the similarity is a value 
between 0 and 1. If a category has a large number of images, 
the matrix will be huge. Therefore, in accordance with the 
present invention, a sparse matrix is used and the similarity 
value is stored only if it is greater than a threshold. 

Fourth, clustering is described as follows in conjunction 
with FIG. 6. Each image is considered to be a node in a 
graph, shown in FIG. 6 as a circle. A link, shown in FIG. 6 
as a line connecting nodes, exists between two nodes in the 
graph if the similarity between the two corresponding 
images is greater than a threshold. Initially, the threshold is 
set at 0.1. Then the connected components in the graph are 
found, A connected component is a set of nodes such that 
each node is linked to at least one other member of the set, 
shown by the dotted rectangles in FIG. 6. The set of nodes 
is selected to be as large as possible, i.e. all of the connected 
nodes are selected to be part of the connected component. 
The set of nodes in a connected component form a cluster. 
This single-link cluster formation technique has been exten- 
sively used in information retrieval and found to be one of 
the most appropriate. See Van-Rijsbergen, "Information 
Retrieval," Butterworths, 1979, Chapter 3 for details. Then, 
if a cluster contains a large number of documents and the 
threshold value is less than the maximum possible value of 
1.0, the threshold value is increased by 0.05 and the above 
clustering process is repeated until a desirable number of 
documents is contained in the cluster. 

Fifth, cluster topics are determined. The clusters are 
represented in the QRVE by the main topic of the clusters. 
The topic of a cluster is determined by the words that appear 
in most of the images of the cluster. Since adverbs, 
prepositions, etc. are not very interesting, they are not 
allowed to be cluster topics. Wordnet can be used to deter- 
mine the figure of speech of a word. See Miller, "Wordnet: 
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A Lexical Database for English" Communications of the 
ACM, 38(11):39-41, November, 1995. Moreover, if a word 
appears as a topic in many clusters, it is removed as a topic 
word. 

Grouping by keywords is useful to semantic ally organize 5 
the images retrieved for an image search, as shown in FIG. 
7(a). Each cluster is represented by a glyph and the main 
topics of the cluster are shown at the bottom of the glyph. 
From the cluster topics, the user gets an idea of the types of 
images that will be present in the cluster. For example, the 1Q 
cluster with the topics "burma", "irrawady", etc. will have 
pictures of Burma (the Irrawady is a river in Burma) as 
shown in FIG. 7(b), where FIGS. 7(a) and 1(b) together 
occupy a window. Besides issuing another search with a 
retrieved image from one of the clusters, the user can also 5 
refine the search by looking at all the images of a cluster. 
Since the clusters are formed by text analysis, by refining, 
text and image searching are smoothly integrated. Even if 
the user does not combine text and image searching during 
the querying phase, they are combined during the browsing 2Q 
phase. It should be noted that text clustering is also useful to 
organize the results of a keyword search if many images are 
retrieved. 

Yet another clustering option provided in the QRVE is by 
image composition. During image indexing, AMORE 25 
extracts the visual characteristics of the images automati- 
cally for content-based image retrieval. The present inven- 
tion focuses on the compositional similarity of the objects in 
the images during clustering. To classify the images based 
on the composition of the objects, two steps are required. 30 
First, the objects must be extracted from the images. To do 
this, edges are detected from the image. This edge image is 
used to extract the objects from the image using color, 
location and visual features. 

FIG. 8 explains the object extraction procedure with the 35 
help of four sets of images. The left images of the sets are 
the original images, the center images are the edge images 
and the right images represent the extracted objects from the 
images. As seen from the right images, the airplane from the 
image 01 and the yacht from image 02 were successfully 40 
extracted. The algorithm also extracted the clouds and sky 
from image 01 and clouds, sea and sky from image 02. 
However, the person in image 03 was divided into two 
regions because of the different color elements. In image 04, 
the two persons were merged into one region because of the 45 
closeness of the color, texture and position of each person. 
These gaps between the visual characteristics and semantic 
meaning of the image can lead to an image classification that 
is not expected by the users. In order to reduce region 
division and region merge effects and provide intuitive 50 
results, the edge images are used directly to evaluate the 
boundary of the objects. 

Once the objects are extracted from the images, the 
second step is to evaluate the composition of the objects and 
cluster images based on the evaluation. Usually, there exist 55 
many different objects in one image and sometimes it is hard 
to extract the objects correctly. In addition, the objects 
contained in the images have a wide variety of shapes of 
different sizes. Due to these reasons it is very difficult to 
determine the cluster automatically, based upon the distri- 60 
bution of the visual attributes. Therefore, typical primary 
shapes like ellipses, rectangles and triangles of different 
sizes and aspect ratios are defined as templates and are used 
as clustering seeds to make a classification. Some of these 
templates are shown in FIG. 9(a). 55 

The clustering process uses a comparison algorithm to 
match the edge images with the input templates as shown in 



FIG. 9(b). The objects in the images do not usually have the 
same shape as the templates. There might exist some 
unevenness and the size might be different. In order to allow 
some flexibility, the algorithm considers the local correlation 
among the data during the comparison. The details of the 
comparison algorithm can be found in related application 
Ser. No. 09/064,061, the contents of which are incorporated 
herein by reference, entitled "Method and Apparatus for 
Image Querying Using Object Boundaries" to K. Hirata and 
assigned to the assignee of the present application. 

To achieve location independence, several sets of images 
are created from the templates by shifting the objects and the 
comparison algorithm is called for each as shown in FIG. 
9(b). Since the algorithm also allows for noise, images are 
created for every 15% shifting, for example. For each image 
of a category, a score determines its similarity to a template 
image of a given size and aspect ratio. If the score is greater 
than a threshold, the image belongs to the corresponding 
cluster. 

An example of image clustering is shown in FIGS. 10(a) 
and 10(6). Each image cluster is represented by a glyph in 
FIG. 10(a). The template image for the cluster is shown on 
the right and the size of the template that corresponds to a 
cluster is displayed at the bottom of the cluster's glyph. The 
size, shape and color of the glyphs are determined by the 
criteria described above. The user can click on a glyph to 
find the retrieved images containing objects similar to the 
template image of the given size and aspect ratio. It is also 
possible to retrieve all images for a cluster. For example, the 
top 6 images having an object similar to a 24x72 ellipse are 
shown in FIG. 10(6). Since querying is by keyword and the 
clustering is by image shape similarity, this gives the user 
another way of combining text and image searching. It 
should be noted that image clustering could also be used to 
organize the results of an image search. 

Thus, in accordance with the present invention, a user of 
a search engine is provided with a method and apparatus for 
providing a user with an interface for refining the results of 
an original query by clustering the results of the original 
query in various ways. 

Other modifications and variations to the invention will be 
apparent to those skilled in the art from the foregoing 
disclosure and teachings. Thus, while only certain embodi- 
ments of the invention have been specifically described 
herein, it will be apparent that numerous modifications may 
be made thereto without departing from the spirit and scope 
of the invention. 

What is claimed is: 

1. A method for refining a first query applied to a database 
of images, the method comprising the steps of: 

selecting one of a plurality of modes of clustering results 
of the first query, wherein the modes of clustering 
include clustering by one of text, image and Universal 
Resource Locator (URL); and 

retrieving cluster groups containing the results of the first 
query, the cluster groups having been clustered in 
accordance with the selected mode of clustering, 
wherein when the selected mode of clustering is clus- 
tering by text, the cluster groups have been formed in 
a preprocessing procedure comprising the steps of: 

(a) representing each of the images in the database in a 
vector space model, where V represents a unique set 
of keywords assigned to the images, where each of 
the images is represented by a vector of length |V|; 

(b) calculating a similarity value between each pair of 
the images, thereby forming a similarity matrix, 
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wherein the similarity value is measured by the 
cosine of the angle between the vector of the respec- 
tive images, the similarity value being further nor- 
malized to a value between 0 and 1; 

(c) storing the similarity value in the similarity matrix 5 
only when the similarity value is greater than a first 
threshold value; 

(d) clustering the images based on the normalized 
similarity value; and 

(e) determining a topic for each of the cluster groups; 10 
and 

the method for refining a first query filer comprising 
displaying the cluster groups, each of the cluster groups 
being visually represented by a graphical element; and 

refilling the results of the first query using members of the 15 
cluster groups as elements of a refined query. 

2. The method according to claim 1, wherein the database 
of images is a multimedia database. 

3. The method according to claim 1, wherein te database 

of images is on the World Wide Web. 20 

4. The method according to claim 1, wherein clustering by 
image includes clustering by color. 

5. The method according to claim 1, wherein clustering by 
image includes clustering by primary shape. 

6. The method according to claim 1, wherein the length 25 
| V| is calculated so that the absence of one of the keywords 

is indicated by a 0 and the presence of one of the keywords 
is indicated by a positive number representing the impor- 
tance of the keyword, the frequency of assignment of the 
keyword in the image. 30 

7. The method according to claim 1, wherein the prepro- 
cessing step further comprises the steps of: 

(a) determining linking nodes, where each of the images 
is represented by a node, wherein nodes are linking 
nodes when the normalized similarity value between 35 
the pair of images is greater than a second threshold 
value; 

(b) determining a set of connected components, the set of 
connected components being a set of the nodes where 
each of the nodes is linked to at least one other member 40 
of the set of nodes, the set of nodes in a connected 
component forming a cluster; and 

(c) repeating steps (a) and (b) by increasing the second 
threshold value when the cluster contains a large num- 
ber of the images and the second threshold value is less 
than a maximum value of 1.0. 

8. The method according to claim 1, wherein the step of 
determining a topic for each of the cluster groups comprises 
determining the keywords which are assigned to most of the 
images of the cluster. 

9. The method according to claim 1, wherein the step of 
refining further comprises the steps of: 

(a) selecting the graphical element representing a desired 
cluster group; 55 

(b) displaying the members of the selected cluster group; 
and 

(c) selecting one of the displayed members as one of the 
elements of the refined query. 

10. The method according to claim 1, wherein, when the 60 
selected mode of clustering is clustering by image, the 
cluster groups have been formed in a preprocessing proce- 
dure comprising the steps of: 

(a) detecting edge images of each of the images in the 
database; 65 

(b) comparing the edge images of each of the images with 
primary shape cluster templates; and 
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(c) grouping the images into cluster groups by primary 
shape based upon similarity between the edge images 
of each of the images and the primary shape cluster 
templates. 

11. The method according to claim 10, wherein the step of 
comparing compensates for unevenness in the edge images, 
small size differences between the edge images and the 
primary object cluster templates and shifting of objects in 
the images. 

12. The method according to claim 10, wherein the step 
of grouping further comprises grouping into cluster groups 
by different sizes of the primary shape cluster templates. 

13. The method according to claim 12, wherein the step 
of refining further comprises the step of selecting the graphi- 
cal element representing a desired cluster group indicating a 
desired size of a desired primary shape cluster template. 

14. The method' according to claim 1, wherein when the 
selected mode of clustering is clustering by URL, the cluster 
groups have been formed by grouping the results by web 
sites from which the results come, wherein each web site 
represents one of the cluster groups. 

15. The method according to claim 14, wherein a direc- 
tory structure is shown for the web site represented by the 
graphical element representing a selected cluster group. 

16. The method according to claim 14, wherein the step 
of refining further comprises the steps of: 

(a) selecting the graphical element representing a desired 
cluster group; 

(b) displaying the members of the selected cluster group; 
and 

(c) selecting one of the displayed members as one of the 
elements of the refined query. 

17. The method according to claim 1, wherein the graphi- 
cal element is a first shape when it is not selected and the 
graphical element is a second shape when it is selected. 

18. The method according to claim 1, wherein a size of the 
graphical clement is proportional to the number of the 
members in the cluster group represented by the graphical 
element. 

19. The method according to claim 1, wherein the graphi- 
cal element is comprised of a first color component and a 
second color component, where the first color component is 
proportional to a maximum number of desired keywords in 
the members of the cluster group represented by the graphi- 
cal element and where the second color component is 
proportional to a maximum image similarity between the 
first query and the members of the cluster group represented 
by the graphical element. 

20. A system for refining a first query applied to a database 
of images, the system comprising: 

a first computer, including a display; and 

a second computer executing an application program for 
providing a user of the computer with an option of 
selecting one of a plurality of modes of clustering 
results of the first query, wherein the modes of clus- 
tering include clustering by one of text, image and 
Universal Resource Locator (URL); wherein the first 
query is executed and the cluster groups containing the 
results of the first query are retrieved, the cluster groups 
having been clustered in accordance with the selected 
mode of clustering; wherein, the cluster groups are 
displayed on the display, each of the cluster groups 
being visually represented by a graphical element; 
wherein the user may refine the results of the first query 
using members of the cluster groups as elements of a 
refined query; and wherein, when the selected mode of 
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clustering is clustering by text, in the application (b) displaying the members of the selected cluster group 

program, the cluster groups have been formed in a on the display; and 

preprocessing procedure by: (c) selecting one of the displayed members as one of the 

(a) representing each of the images in the database in a elements of the refined query. 

vector space model, where V represents a unique set 5 31. The system according to claim 20, wherein, when the 

of keywords assigned to the images, where each of selected mode of clustering is clustering by image, in the 

the images is represented by a vector of length |V|; application program the cluster groups have been formed in 

(b) calculating a similarity value between each pair of preprocessing procedure by: 

the images, thereby forming a similarity matrix, 0) detecting edge images of each of the images in the 

wherein the similarity value is measured by the 10 database; 

cosine of the angle between the vector of the respec- ( b ) comparing the edge images of each of the images with 

tive images, the similarity value being further nor- primary shape cluster templates; and 

malized to a value between 0 and 1; ( c ) grouping the images into cluster groups by primary 

(c) storing the similarity value in the similarity matrix based u P on similarity between the edge images 
only when the similarity value is greater than a first 15 of cach of the and the P nmar V sha P c cluster 
threshold value, _. tc *P lates ; . 

(d) clustering the images based on the normalized 32 ' ™ e s y stem t acc f dia S to claun 31 ' the com * 
• -i - t i j paring compensates for unevenness in the edge images, 

similarity value; and smaU &ize differences betwe cn the edge images and the 

(e) determining a topic for each of the cluster groups afld g of ^ 

21. The system according to claim 20, wherein the first 20 ^ e& J r & j 

computer and the second computer are the same computer. -* ™ ' , . . 

„ p _ , / , . - A . 33. The system according to claim 31, wherein the group - 

22. The system according to claim 20. wherein the first . • . i . u j«- * * r *£ 

t . , . j *u j * • mg groups into cluster groups by different sizes of the 

computer is a client computer and the second computer is a • \_ i * * i * 

- , , . _j , . . . . . r pnmary shape cluster templates, 

host computer connected to the client computer. * 34 ^ e ^ ^ ^ daim 33 wherein ^ 

23. The system according to claim 20, wherein the data- 25 refloed b ^ deling the graphical element represent- 

base of images is a multimedia database. . j • j i . j • j • r 

*j . j. , * . t_ * *i_ j mg a desired cluster group indicating a desired size of a 

24. The system according to claim 20, wherein the data- , . , . . , e . . , , 

. f . 7 - .i "\i7 ij , ir . « 7 u desired primary shape cluster template. 

base .of] images is on the World Wide Wfeb. 35. Tie system according to claim 20, wherein when the 

25. THe system according tc .clam, 20, wherein clustering sel6cted ' f c]ustcri UR 

by miage mcludes clustering by color. 30 , & . , & / f ' 

'-..Jr. . . i.- apphcation program, the cluster groups have been formed by 

26. The system according to claim 20, wherein clustering £ £ ^ web ^ £ Qm whicb ^ 

by image mcludes clustering by pnmary shape. each w ' b ^ nU one of lhe clusler 

27. The system according to claim 20, wherein the length 

I VI is calculated so that the absence of one of the keywords Sr ^^ , j-.i -icu j * 

! ! , r\ j .1 e r*u i j „ , 36. The system according to claim 35, wherem a directory 

is indicated by a 0 and the presence of one of the keywords 35 4 . , , / , , iL , . t J 

. . A , . ... r . * structure is displayed on the display for the web site repre- 

is indicated by a positive number representing the impor- * j u *u u- i i * *• t * j 

^ . L / j ^ * r • A /.i. sented by the graphical element representing a selected 

tance of the keyword, the frequency of assignment of the c i uster group 

ke ^°^ lD ^ imaSC * j* . i • * ' ' *u 37. The system according to claim 35, in the apphcation 

28. The system according to claim 20, wherem in the u <? , , rr 

* * ^ . ,u • ■ f. ■ program, the results are refined by: 

application program, the preprocessing procedure further 40 r * , , , . . . , . , . , 

• (a) selecting the graphical element representmg a desired 

comprises. , 

* t , , . * , . cluster group; 

(a) determimng linking nodes, where each of the images ,. . , t . . . . . 

v 7 . j . j i_ j ! . fb) displaying the members of the selected cluster group 

is represented by a node, wherem nodes are linking v ' ^ dis la ■ and 

nodes when the normalized similarity value between . ° n , e . p ay> . 

the pair of images is greater than a second threshold 45 (c) selecUng one of the displayed members as one of the 

ya j u ^. elements of the refined query. 

„ x , ' . . r , , 38. The system according to claim 20, wherein the graphi- 

(b) determining a set of connected components, the set of ca] elem6nt fa a ^ sh when ft fa not and the 

connected components being a set of the nodes where nical clement ^ a sh wheD a b sclected 

each of the nodes is linked to at least one other member 39 ^ according to cUim 20 , wherein a size of 

of the set of nodes, the set of nodes in a connected ^ elemenl ^ proportiona] t0 the number of the 

component forming a cluster; and members in the cluster group represented by the graphical 

(c) repeating (a) and (b) by increasing the second thresh- element. 

old value when the cluster contains a large number of 40. The system according to claim 20, wherein the graphi- 

the images and the second threshold value is less than 55 ca i element is comprised of a first color component and a 

a maximum value of 1 .0. second color component where the first color component is 

29. The system according to claim 20, wherein a topic for proportional to a maximum number of desired keywords in 
each of the cluster groups is created by determining key- me members of the cluster group represented by the graphi- 
words which are assigned to most of the images of the ca i element and where the second color component is 
cluster. 6Q proportional to a maximum image similarity between the 

30. The system according to claim 20, wherein in the first query and the members of the cluster group represented 
application program, the results are refined by: Dy the graphical clement. 

(a) selecting the graphical element representing a desired 

cluster group; ***** 
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